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The current research employs the use of content analysis to teach research methods concepts among 
students enrolled in an upper division research methods course. Students coded and analyzed Jimmy 
Buffett song lyrics rather than using a downloadable database or collecting survey data. Students’ 
knowledge of content analysis concepts increased after a lecture on the topic of content analysis, but 
they further improved after participating in the song coding, data cleaning, and writing of results. 
Additionally, students reported high satisfaction with the project and believed it was an interesting 
and enjoyable technique for learning about research methods. We provide suggestions for 
incorporating similar data collection activities in undergraduate research methods courses. 


Over the past few decades there has been a push to 
reformulate the education of undergraduate students in 
research intensive institutions. Proponents for this 
reformulation argue that students are constantly failed 
by universities that separate undergraduate teaching and 
advanced research (Strum Kenny, 1999). Arguably, the 
better educational program is one that combines 
teaching and tools for research so that students have the 
ability to analyze and contribute to research (Jenkins & 
Zetter, 2003). Many departments include research 
methods courses in their required curriculum; therefore, 
it is important to consider ways these classes might 
better facilitate critical thought and knowledge of how 
to perform research. Because it is clear that the 
connection between the teaching of research methods 
and the ability to perform research is not automatic, the 
formation of a research and teaching nexus is critical 
(Jenkins & Zetter, 2003). Jenkins, Breen, Lindsay, and 
Brew (2002) suggest utilizing factors from the students’ 
social world as one strategy for linking teaching and 
research at the undergraduate level. The current article 
is an assessment of one such training exercise aimed at 
teaching undergraduate students the research practice of 
content analysis on a common factor from students’ 
social world — song lyrics. 

The use of activities in a classroom setting 
stimulates student interest in learning and provides a 
number of advantages (Bernstein, 1999). First, 
participation in classroom activities allows students to 
become actively engaged in learning new concepts. 
Incorporating activities into the classroom provides 
students with another method of learning (i.e., learning 
by doing) in addition to the traditional method of 
passively listening to a lecture. Second, activities often 
change the traditional pace of the classroom such that 
students are regularly challenged to pay attention. 
Third, participation in classroom activities has been 
shown to provide both students and instructors with 
more enjoyable methods of facilitating learning. 
Marek, Christopher, and Walker (2004) found that 


incorporating an active-learning approach to teaching 
research methods results in greater learning success for 
students. It is clear that implementing hands-on 
activities in the classroom can be beneficial for teaching 
new concepts. In addition to the use of classroom 
activities, topics perceived by students to be interesting 
have been shown to be more effective in facilitating 
learning than topics perceived by students to be 
monotonous (U.S. Department of Education, 1987). 
Interesting activities and topics tend to capture 
students’ attention, hold their attention longer than 
uninteresting activities, and activate students’ interest in 
learning. Additionally, several instructors note that 
implementing interesting classroom activities results in 
higher academic performance by students (Garcia & 
Garcia, 2004; Rajecki, 2002). 

Classroom activities involving content analysis 
have been shown to be an interesting and educational 
means of teaching research methods. For example, 
Rajecki (2002) describes the benefits of analyzing the 
content of newspaper personal advertisements, and 
Carpenter (1998) illustrates an activity comprised of 
analyzing the content of articles portraying social 
stereotypes. Although these two studies demonstrate 
success with content analysis activities, a quick review 
of recently published introductory level research 
methods textbooks shows a relative inattention to this 
research method. Examining five different textbooks, 
we found between 0% and 3% (M = 1.5%) of the text 
pages offered information on content analysis versus 
6.5% to 11% (M = 6.4%) of the text pages for survey 
research (Bachman & Schutt, 2007; Hagan, 2006, 2007; 
Maxfield & Babbie, 2008; Schutt, 2006). 

In accordance with the suggestion by Jenkins and 
Zetter (2003), we implemented an exercise created to 
help teach content analysis to an undergraduate 
research methods class at a large research university. 
The project required students to code song lyrics 
written by Jimmy Buffett as a way to incorporate the 
students’ social world, increase understanding of 
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content analysis specifically, and increase 
appreciation for research methods generally. 

Method 

Participants 

Twenty-five undergraduate students who were 
registered for a research methods class (16 women 
and 9 men) at a large Southeastern university 
participated in the current class project and all 
phases of testing. Eighty percent of the students who 
participated in all three waves were either juniors or 
seniors; the other 20% were sophomores. An 
additional 15 students were used as a control group. 

Materials 

At three different points, each participant 
completed a 12 question knowledge test that 
included 11 multiple choice questions about content 
analysis and research methods. For example, one 
question was, “If a study was coding latent content 
rather than manifest content, then we would expect 

to see inter-rater reliability” (correct answer 

option: lower). Another question was, “A content 
analysis researcher interested in a newspaper’s 
commitment to the community might operationalize 

that concept in terms of ” (correct 

answer option: how many different bylines appear on 
stories about local government meetings). The 
twelfth question was open-ended and asked the 
students to develop a content analysis research 
project on their own to address a specific research 
question. The question was, “A certain researcher 
was interested in high school friendships. Describe a 
content analysis study addressing this issue.” At the 
end of the study, students also completed an eight 
question satisfaction measure similar to that used by 
Marek and colleagues (2004). Example questions 
from this measure included the following: “Overall, I 
would recommend including a class project similar 
to this one” and “I enjoyed this class project.” 
Students answered on a scale from 1 (strongly 
disagree) to 5 (strongly agree) with higher scores 
indicating more satisfaction with the project. 

Procedure 

Right before a midsemester break, students took 
the knowledge test to obtain a baseline measure. 
When the students returned from the break, the 
professor lectured on content analysis as a research 
technique. The students took the knowledge test 
again during the following class to measure their 
postlecture knowledge. 


The class then started working on the content 
analysis project. We chose Jimmy Buffett song 
lyrics because Buffett’s career has spanned more 
than 30 years, his lyrics are readily available on his 
website, most students knew of him, and researchers 
have previously addressed the content of his music 
without doing a formal systematic study of his lyrics 
(Bowen, 1997; Mihelich & Papineau, 2005). 
Students developed research questions and 
hypotheses based largely on the articles that had 
addressed Buffett’s music. For instance, Mihelich 
and Papineau (2005) note that Buffett’s career 
changed in 1984 when he obtained corporate 
sponsorship; therefore, students hypothesized that 
the content of his music changed after 1984. General 
knowledge of his songs led students to hypothesize 
that the change would be mostly reflected in lyrics 
that mention alcohol, drugs, and deviant behavior. 

We randomly assigned each student the lyrics 
from seven Jimmy Buffett songs resulting in two 
separate coders who were responsible for coding 
each song. The students recorded information such 
as year the song was released, whether the song 
charted, and on how many albums the song 
appeared. The students coded each of the songs for 
manifest content such as whether alcohol was 
mentioned in the title of the song, number of 

references to criminal activity, and number of 

references to alcohol and drugs in the lyrics. 

Students also coded for latent content such as 

overall theme of the song (e.g., love, humor, and 
social protest) and whether the song glamorized 
alcohol consumption. Comparisons between the 
coders revealed high consistency for the manifest 
content and lower consistency on the latent content. 
We used the inconsistency between coders as a 
class demonstration about intercoder reliability and 
the importance of clear operational definitions in 
research. After we resolved the inconsistencies 
between the coders, we removed duplicate songs so 
that each song represented one unit of analysis. 

In order to demonstrate the concept of 
inferential statistics, we selected a sample of the 
songs on which the students performed univariate 
and bivariate analyses to test their hypotheses. 
Acquiring the population of songs from which the 
sample was selected allowed for a demonstration of 
the concepts of sampling techniques and hypotheses 
testing. The students wrote their final class papers 
based on the sample of songs that included the 
following sections: hypotheses, methods, results, 
discussion, and conclusions. On the same day their 
papers were due, the students took the knowledge 
test a third time and also took the satisfaction 
measure. The entire process took five weeks from 
the initial pretest to the final posttest. 
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Results 

Student Learning 

Repeated measures ANOVA revealed that student 
knowledge of content analysis concepts significantly 
increased between each of the three tests, F( 2, 48) = 
55.85, p < .001, partial eta 2 = .70. This increase was 
linear ( F (1, 24) = 1 12.66, p < .001), with scores before 
the lecture averaging 36% correct on the 11 -item 
multiple choice portion of the knowledge test. After the 
lecture the average score was 56% correct, and after the 
music lyrics project the average was 72% (See Table 1). 
The difference was also noticeable on the open-ended 
portion of the test. Although the open-ended question 
was frequently left blank by the students (44% blank in 
pre-test, 36% blank in test after lecture, and 40% blank 
in post- test after completion of project), no student who 
attempted to answer this question at the baseline (before 
the lecture) accurately described an appropriate content 
analysis project. Students were asked to describe a 
content analysis study addressing high school 
friendships, but in the pre-lecture and post-lecture 
conditions they often described surveys or 
observational research methods. For example, one 
student wrote, “Analyze two people who share a 
friendship over a long period of time.” After the lecture 
(the second test time) all of those who tried were either 
completely wrong in their descriptions or they made 
fairly serious omissions. For example, one student 
wrote, “You could design a study that looked at the 
type of seating arrangements in a classroom and at 
lunch if they weren’t set by a teacher.” After the class 
project (the third test time), all but one student who 
attempted this question accurately described an 
appropriate content analysis project. For example, one 
student described in great detail a project that involved 
reading and coding the messages people wrote in 
yearbooks. 

To ensure that the increase in scores was not 
simply due to testing effects, we administered the same 
tests with the same time delays (same five week 
schedule that included a mid-semester break) and the 
same lecture to a control group in a different research 
methods class. The control group had no significant 
differences between the three testing times (percentage 
correct = 33%, 29%, and 29% respectively) ( F (2, 28) = 
.55, p = .58, partial eta 2 = .04). In addition, no student 
in the control group correctly answered the open-ended 
question in any of the three test times. The slight 
decrease in scores after the lecture in the control group 
may be attributed to the fact that the students were not 
as interested as the experimental group in the topic 
because their final project was not going to involve 
content analysis. As discussed above, content analysis 
is unfamiliar to most undergraduates and the slight 


differences in mean scores are likely due to differences 
in student guessing. The control group scores do 
demonstrate that the improvement in scores for the 
experimental group was not due to repeat testing 
effects. 

Student Satisfaction 

Students’ satisfaction as measured by an 8-item 
satisfaction scale was quite high (M = 4.00, SD = 0.61, 
on a scale of 1 to 5 with 1 being Strongly Disagree and 
5 being Strongly Agree and higher scores indicating 
more satisfaction). Overall, the students found the 
project to be enjoyable and useful in their 
understanding of research methods and content 
analysis. The students were also asked to provide their 
opinions of the project. A few examples follow: 

• “[The project was] much more interesting and 
easy to get into than a random data set forced 
on us,” 

• “I believe it helped me understand the 
concepts better than to just have lectures on 
the material,” 

• “I think the project was useful because we 
were involved in every step, so we could see 
what was going on,” 

• “[The project] was excellent and one of the 
best class projects I’ve done during my three 
years in school,” 

• and “It was vastly more educational to collect 
and code the data than just read about it in a 
textbook.” 

Discussion 

Students reported learning from and enjoying the 
content coding project, and their knowledge of content 
analysis improved throughout the project. Students had 
a basic increase in knowledge after a lecture on the 
topic of content analysis, but their ability to apply that 
knowledge and develop their own content analysis 
project was better solidified after they had done the 
class project. Arguably, the scores still only reached an 
“average” level (72%), but that was likely due to the 
fact that the knowledge test was intentionally difficult, 
as demonstrated by the extremely low scores at 
baseline. Additionally, the students were not told that 
they would be taking the test and, therefore, would not 
have studied the material. The scores represent what 
the students knew without actively studying for an 
exam on the topic. 

The open-ended question included in the 
knowledge test had a fairly low response rate 
(approximately 60%) even in the posttest condition. We 
attribute the high rate of non-completion to the 
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Table 1 


Mean Scores for Experimental and Control Conditions 


Measure 1,2 

Mean 

SD 

Control Group (n =15) 

Pretest (before lecture) 

3.63 

1.80 

Test (after lecture) 

3.19 

1.93 

Post (after delay) 

3.19 

1.74 

Experimental Group (n = 25) 

Pretest (before lecture) 

3.96 

1.62 

Test (after lecture) 

6.16 

1.89 

Post (after delay for project) 

7.92 

1.59 

Satisfaction with Project 

4.0 

.61 


1 A score of 1 1 was the highest possible correct on the pretest, test, and posttest. A score of 0 was the lowest 
possible score. 

2 The satisfaction score could range from 1 to 5 with higher values indicating greater satisfaction. 


extra effort this question required over the effort for 
the other questions on the test. Similar to other non- 
required and non-graded measures (e.g., teacher 
evaluations), the students likely decided the open- 
ended question was too much effort. Most 
importantly, we did see a great deal of improvement 
in ability to develop their own research questions and 
methodology for those students who did choose to 
complete the open-ended question. 

Although not part of the formal knowledge test 
described above, the concepts of hypothesis testing, 
units of analysis, populations, and samples were also 
better understood by students. In past semesters 
when students performed statistical analyses and 
wrote papers based on a large database provided to 
them by the instructor, these concepts were often 
difficult and the questions concerning these concepts 
were frequently missed on the exams. In contrast, 
during the semester with the content analysis project, 
the students’ exam scores demonstrated that they 
achieved a better understanding of these general 
research methods and statistical concepts. 

Content analysis as a research technique is 
underrepresented in research methods texts and 
courses, but particularly useful because it easily 
demonstrates to students how information in their 
daily life can be sources of research data. It is also an 
easy way for students to collect their own data 
without any of the Institutional Review Board 
difficulties that might be present when using a 
survey technique. Song lyrics are a practical source 
of content because most are readily available online 
and are easily assignable to individual students to 
code. 

Other research methods classes could replicate 
the current project using the same Jimmy Buffett 
lyrics or lyrics from another musician, musical 
group, or musical genre. For instance, Cole (1971) 


content analyzed top- 10 singles from each year of the 
1960s. He coded each song for mood, love-sex, 
religion, violence, and social protest. A research 
methods class could similarly examine multiple 
decades and multiple musical genres. Additional 
content coding projects could easily be done on 
magazine articles, newspaper articles, television 
shows, or any other readily available social artifacts. 
To capitalize on the popularity of reality television 
shows and their availability online, a class project 
could have students code episodes of a reality show 
for instances of racial and gender stereotyping or 
stereotype threat. By bringing these elements from 
our daily world into the research methods classroom 
we will be able to convey to our students that 
conducting research is not only accessible to them, 
but that it is relevant to their daily lives. 
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